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Program Description^ 

DreamBox Learning is a supplemental online mathematics program 
that provides adaptive instruction for students in grades K-5 and 
focuses on number and operations, place value, and number sense. 

The program aims to individualize instruction for each student with 
millions of unique paths through the curriculum intended to match 
each student’s level of comprehension and learning style. The cur- 
riculum is based on the National Council of Teachers of Mathematics 
(NCTM) standards. 

Research^ 

The What Works Clearinghouse (WWC) identified one study of 
DreamBox Learning that both falls within the scope of the Elementary 
School Mathematics topic area and meets WWC evidence standards. 

This study meets standards without reservations and included 557 
elementary school students in kindergarten and first grade in three 
charter schools in San Jose, California. 

The WWC considers the extent of evidence for DreamBox Learning 
on the math performance of elementary school students to be small 
for the mathematics achievement domain, the only domain specified 
in the review protocol. 

Effectiveness 

DreamBox Learning was found to have potentially positive effects on mathematics achievement for elementary 
school students. 


Table 1. Summary of findings^ 




Improvement index 

(percentiie points) 




Outcome domain 

Rating of effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Mathematics achievement 

Potentially positive effects 

+4 

na 

1 

557 

Small 


na = not applicable 
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Program Information 

Background 

DreamBox Learning was developed and is distributed by DreamBox Learning, inc. Address: 305 108th Avenue NE, 
2nd Fioor, Beilevue, WA 98004. Email: schools@dreambox.com. Web: http://www.dreambox.com. Telephone: 

(877) 451-7845. 

Program details 

The DreamBox Learning curriculum is made up of more than 720 online lessons for two grade levels: primary les- 
sons for grades K-2 and intermediate lessons for grades 3-5. Students complete lessons by playing math games 
and solving puzzles using DreamBox Learning’s virtual manipulatives. As students interact with the program, the 
lesson sequence, difficulty, pace, and number of hints are adapted in real time. DreamBox Learning can be used 
In a range of Instructional settings including the classroom, small groups, or students using the program indepen- 
dently. Reporting tools allow teachers and administrators to monitor student progress. DreamBox Learning recom- 
mends students spend at least 90 minutes a week on the program. 


Cost 

A license to use DreamBox Learning for individual students, classrooms, or grades costs $20 per student per year. 
A license for a school, which covers an unlimited number of students, costs $7,000 per year. 
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Research Summary 

The WWC identified 1 1 studies that investigated the effects of Table 2. Scope Of reviewed research 

DreamBox Learning on the math performance of eiementary 
school students. 

The WWC reviewed one of those studies against group design evi- 
dence standards. This study (Wang & Woodworth, 2011) is a ran- 
domized controlled trial that meets WWC evidence standards without 
reservations. The study is summarized in this report. The remaining 10 studies do not meet WWC eligibiiity screens 
for review in this topic area. Citations for aii 1 1 studies are in the References section, which begins on p. 5. 

Summary of studies meeting WWC evidence standards without reservations 

Wang and Woodworth (2011) examined the effects of DreamBox Learning on 557 kindergarten and first-grade 
students in three charter eiementary schools located in San Jose, California. Within each school, all students in 
grades K and 1 were randomly assigned to either an intervention or comparison group. The intervention group 
received DreamBox Learning supplemental mathematics instruction every day, while the comparison group 
received additional literacy instruction. Among the randomly assigned students, some students identified by the 
school as low-achieving received after-school programming known as Response to Intervention (Rtl). Those stu- 
dents also received DreamBox Learning instruction in their after-school Rtl programming, regardless of Intervention 
status. For intervention group students who participated in Rtl, the after-school DreamBox Learning instruction was 
in addition to the DreamBox Learning instruction provided during the school day. 

Summary of studies meeting WWC evidence standards with reservations 

No studies of DreamBox Learning met WWC evidence standards with reservations. 


Grade 

K-1 

Delivery method 

Individual 

Program type 

Supplement 
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Effectiveness Summary 

The WWC review of DreamBox Learning for the Elementary School Mathematics topic area includes student out- 
comes in one domain: mathematics achievement. The findings beiow present the authors’ estimates and WWC- 
calculated estimates of the size and statisticai significance of the effects of DreamBox Learning on eiementary 
schooi students. For a more detaiied description of the rating of effectiveness and extent of evidence criteria, 
see the WWC Rating Criteria on p. 12. 

Summary of effectiveness for the mathematics achievement domain 

One study that meets WWC standards without reservations reported findings in the mathematics achievement 
domain. 

Wang and Woodworth (2011) reported a statisticaily significant positive difference between students in the Dream- 
Box Learning and comparison groups on the Measures of Academic Progress (MAP) overaii math score. However, 
this anaiysis excludes cases whose outcomes (posttest math scores) were classified as outliers by the authors. The 
authors also found a statistically significant positive difference between the groups in an anaiysis that included the 
full sample, as reported in response to an author query. The WWC rating of the study was based on the analysis 
that included the full sample. 

Thus, for the mathematics achievement domain, the study showed a statistically significant positive effect. This 
results in a rating of potentially positive effects, with a small extent of evidence. 


Table 3. Rating of effectiveness and extent of evidence for the mathematics achievement domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of a positive effect with no 
overriding contrary evidence. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the mathematics 
achievement domain was positive and statistically significant. 

Extent of evidence 

Criteria met 

Small 

One study that included 557 students in three schools reported evidence of effectiveness in the mathematics 
achievement domain. 
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Study that meets WWC evidence standards without reservations 

Wang, H., & Woodworth, K. (2011). Evaluation of Rocketship Education’s use of Dream Box Learning’s online 
mathematics program. Menlo Park, CA: SRI International. Retrieved from http://www.dreambox.com 

Additional source: 

Wang, H., & Woodworth, K. (2011). A randomized controlled trial of two online mathematics curricula. 

Evanston, IL: Society for Research on Educational Effectiveness. 

Studies that meet WWC evidence standards with reservations 

None. 

Studies that do not meet WWC evidence standards 
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Studies that are ineligible for review using the Elementary School Mathematics Evidence Review Protocol 

DreamBox Learning, Inc. (2012). Case study: Accelerating learning: Achieving a 5.5 percentile point gain in 16 
weeks. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is ineligible for review 
because it does not use a comparison group design or a single-case design. 

DreamBox Learning, Inc. (2012). Case study: Building number sense: Tailoring math instruction to individual student 
needs. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is ineligible for review 
because it does not use a comparison group design or a single-case design. 

DreamBox Learning, Inc. (2012). Case study: Embracing blended learning: Providing a world-class education to 
low-income minority students. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is 
ineligible for review because it does not use a comparison group design or a single-case design. 

DreamBox Learning, Inc. (2012). Case study: From good to great: Top performing school district boosts math 
scores. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is ineligible for review 
because it does not use a comparison group design or a single-case design. 

DreamBox Learning, Inc. (2012). Case study: Impactful math intervention: From the 1st national percentile to the 
54th in one year. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is ineligible for 
review because it does not use a comparison group design or a single-case design. 

DreamBox Learning, Inc. (2012). Case study: Making AYR in math: A Title 1 school transformation. Bellevue, WA: 
Author. Retrieved from: http://www.dreambox.com The study is ineligible for review because it does not use a 
comparison group design or a single-case design. 

DreamBox Learning, Inc. (2012). Case study: Managing budget cuts: Cost-effective instruction, assessment and 
math intervention. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is ineligible for 
review because it does not use a comparison group design or a single-case design. 

DreamBox Learning, Inc. (2012). Case study: Personalizing instruction: Providing individualized learning paths for 
3,300 students. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is ineligible for 
review because it does not use a comparison group design or a single-case design. 

Jorgensen, M. (2011). Results from DreamBox Learning embedded assessment study: Demonstrates 50% increase 
in student proficiency in math. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is 
ineligible for review because it does not use a comparison group design or a single-case design. 

Jorgensen, M. (2011). Results from DreamBox Learning grade 2 assessment study: Math achievement test demon- 
strates 19% increase. Bellevue, WA: Author. Retrieved from: http://www.dreambox.com The study is ineligible 
for review because it does not use a comparison group design or a single-case design. 
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Appendix A: Research detaiis for Wang and Woodworth, 2011 

Wang, H., & Woodworth, K. (2011). Evaluation of Rocketship Education’s use of DreamBox Learning’s 
online mathematics program. Menlo Park, CA: SRI International. Retrieved from 
http://www.dreambox.com 

Tahie A. Summary of findings Meets WWC evidence standards without reservations 


study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Mathematics achievement 557 students +4 Yes 


Setting 

The study was conducted in three Rocketship Education charter schools located in San Jose, 


California. 

Study sample 

The study sample included all kindergarten and first-grade students at the three schools that 
participated in the study, a total of 557 students after attrition from a sample of 583 who were 
randomly assigned. The number of classrooms Included in the study is not specified. Within 
grade levels, students were randomly assigned to either the intervention or comparison groups 
at a 4 to 1 ratio. In the baseline sample, 53% of students were female, 87% were Hispanic, 

81 % were English language learners, 88% were eligible for free or reduced-price meals, 4% 
were classified as special education, and 10% participated in Response to Intervention (Rtl) 
services. 

Intervention 

group 

The experiment was conducted from mid-October through mid-February during the 2010-1 1 
school year. Intervention students were scheduled to receive 20 to 40 minutes of DreamBox 
Learning mathematics instruction per day; usage statistics show that students averaged 21.8 
hours of usage over the course of the study, or approximately 16 minutes per day. Instruc- 
tional sessions were conducted in a computer lab. The authors noted that the low-achieving 
students who were assigned to receive Rtl services were scheduled to receive 45 minutes of 
DreamBox Learning instruction in their after-school Rtl programming, regardless of Interven- 
tion status. For the 42 intervention group students who were assigned to Rtl services, this 45 
minutes was in addition to the DreamBox Learning instruction provided during the school day, 
for a total of 26.5 hours of usage over the course of the study on average. Progress and use 
information provided by the DreamBox Learning software was not used to modify face-to-face 
mathematics instruction for either the intervention or comparison group. 

Comparison 

group 

Students in the comparison condition received no additional mathematics instruction. How- 
ever, they received additional literacy instruction via an online program during the time and in 
the same location as intervention group students using the DreamBox Learning software. The 
1 1 students in the comparison condition who were assigned to Rtl services were scheduled 
to receive 45 minutes of DreamBox Learning instruction in their after-school Rtl programming; 
the authors found that these comparison condition students averaged 5.1 hours of program 
usage over the course of the study. 
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Outcomes and 
measurement 


Support for 
implementation 


The study used math test scores from the MAP assessment developed by the Northwest Eval- 
uation Association (NWEA). The study reports the overall math score, as well as five subtest 
scores, for problem solving, number sense, computation, measurement and geometry, and 
statistics and probability. Scores were scaled using the RIT scale, “which is scaled using the 
Item Response Theory (IRT) and has the same meaning regardless of the grade of the student” 
(as cited in Wang & Woodworth, 201 1 , p. 3). The schools administered the assessment in Sep- 
tember 2010 (pretest) and January/February 201 1 (posttest). For a more detailed description of 
this outcome measure, see Appendix B. 

DreamBox Learning “does not prescribe a specific role for teachers” (Wang & Woodworth, 

201 1 , p. 3). The computer labs In which students received DreamBox Learning instruc- 
tion were run by lab coordinators, noncredentialed hourly staff who played a minimal role In 
instruction. The authors noted that lab coordinators sometimes may have been out of the 
computer lab, at which times the students would be supervised by support staff. 
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Appendix B: Outcome measures for the mathematics achievement domain 


Mathematics achievement 


l\lorthwest Evaluation Association The mathematics test in the MAP assessment developed by the NWEA includes tive subtests: problem solving, 

(NWEA) Measures of Academic Progress number sense, computation, measurement and geometry, and statistics and probability. Scores were scaled 
(MAP): Overall math score using the BIT scale, “which is scaled using the Item Response Theory (IRT) and has the same meaning regard- 

less of the grade of the student” (as cited in Wang & Woodworth, 2011, p. 3). 
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Appendix C: Findings inciuded in the rating for the mathematics achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Wang & Woodworth, 2011^ 

MAP: Overall math score 

Grades K 
and 1 

557 students 

159.00 

(16.60) 

156.20 

(15.10) 

2.80 

0.11 

-f4 

0.04 


Domain average for mathematics achievement (Wang & Woodworth, 2011) 0.11 +4 Statistically 

significant 


Domain average for mathematics achievement across all studies 0.11 +4 Statistically 

significant 


Table Notes: For mean difference, effecf size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for aii students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, refiecting the 
change in an average student's percentiie rank that can be expected if the student is given the intervention. The statisticai significance of the study's domain average was deter- 
mined by the WWC. MAP = Measures of Academic Progress. 

® For Wang & Woodworth (2011), the authors' primary anaiysis uitiiizes an ordinary ieast squares (OLS) approach that inciudes controis for student demographic characteristics, 
pretest score. Response to intervention status, grade ievei, and schooi fixed effects. Flowever, this anaiysis exciudes cases whose posttest scores were deemed to be outiiers by 
the authors. The resuits presented here use OLS estimates and associated p-vaiues based on the fuii sampie, inciuding outiiers, that were suppiied by the authors in response to an 
author query. No corrections for ciustering or muitipie comparisons and no difference-in-differences adjustment were needed. 
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Appendix D: Summary of suppiementai subscaie and subgroup findings for tbe matbematics achievement 
domain 





Mean 

(standard deviation) 

WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Wang & Woodworth, 2011^ 

MAP: Problem Solving 
subtest 

Grades K 
and 1 

553 students 

161.40 

(16.30) 

159.80 

(15.20) 

1.60 

0.07 

-f3 

0.31 

MAP: Number Sense subtest 

Grades K 
and 1 

553 students 

159.60 

(18.90) 

157.00 

(17.20) 

2.60 

0.10 

+4 

0.17 

MAP: Computation subtest 

Grades K 
and 1 

546 students 

163.00 

(20.70) 

158.80 

(19.50) 

4.20 

0.15 

+6 

0.03 

MAP: Measurement and 
Geometry subtest 

Grades K 
and 1 

550 students 

155.50 

(18.30) 

151.80 

(18.10) 

3.70 

0.15 

+6 

0.03 

MAP: Statistics and 
Probability subtest 

Grades K 
and 1 

552 students 

156.30 

(18.90) 

154.10 

(17.60) 

2.20 

0.10 

+4 

0.20 


Table Notes: The supplemental findings presented in this tabie are additionai findings from the studies in this report that do not factor into the determination of the intervention 
rating. For mean difference, effect size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors the 
comparison group. The effect size is a standardized measure of the effect of an infervention on sfudent oufcomes, representing the average change expected for aii students who 
are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, refiecting the change 
in an average student’s percentiie rank that can be expected if the student is given the intervention. 

“ For Wang & Woodworth (2011), the authors' primary anaiysis uitiiizes an ordinary ieast squares (OLS) approach that inciudes controis for studenf demographic characteristics, 
pretest score, Response to intervention status, grade ievei, and schooi fixed effects. Flowever, this anaiysis exciudes cases whose posttest scores were deemed to be outiiers by 
the authors. The resuits presented here use OLS estimates and associated p-vaiues based on the fuii sampie, inciuding outiiers, that were suppiied by the authors in response to an 
author query. A correction for muitipie comparisons was needed and resuifed in WWC-computed criticai p-vaiues of 0.02 for the MAP: Measurement and Geometry suhtest and 0.01 
for the MAP: Computation subtest; therefore, the WWC does not find these resuits to be statisticaiiy significant. No corrections for ciustering and no difference-in-differences adjust- 
ment were needed. 
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Endnotes 

* On February 5, 2014, the WWC modified this report in response to new information provided by the study authors. Following the 
release of the intervention report on December 10, 2013, the WWC received a request from the study authors for another opportunity 
to respond to an author query to which the authors had previously not responded. The authors then provided information that the 
WWC had previously requested— specifically, regression-adjusted impact estimates using the full sample. The authors’ original report 
included regression-adjusted impact estimates only for a sample that excluded students whose outcomes were classified as outli- 
ers by the study authors. Based on the new information provided, the review team updated the characterization of the study finding 
and the intervention rating in this report. The characterization of the study finding changed from “indeterminate effect” to “statistically 
significant positive effect.” The intervention effectiveness rating changed from “no discernible effects” to “potentially positive effects.” 
The WWC has not added studies to the body of evidence or updated the literature search since the December 2013 release of this 
report. 

■' The descriptive information for this program was obtained from publicly available sources: the program’s website 
(http://www.dreambox.com, downloaded December 14, 2012) and Wang and Woodworth (2011). The WWC requests developers 
review the program description sections for accuracy from their perspective. The program description was provided to the developer 
in December 2012, and the WWC incorporated feedback from the developer. Further verification of the accuracy of the descriptive 
information for this program is beyond the scope of this review. The literature search reflects documents publicly available by Decem- 
ber 201 2. 

2 The studies in this report were reviewed using the Evidence Standards from the WWC Procedures and Standards Handbook (version 
2.1) along with those described in the Elementary School Mathematics review protocol (version 2.0). The evidence presented in this 
report is based on available research. Findings and conclusions may change as new research becomes available. 

® For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 12. 
These improvement index numbers show the average and range of student-level improvement indices for all findings across the 
studies. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, 

December). Elementary School Mathematics Intervention report: DreamBox Learning. Retrieved 
from http://whatworks.ed.gov 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 


Study rating 

Criteria 

Meets WWC evidence standards 
without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a weii-implemented RCT. 

Meets WWC evidence standards 
with reservations 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 
attrition that has established equivaience of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statisticaiiy significant positive effects, at least one of which met WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Single-case design 
Standard deviation 


Statistical significance 


Substantively important 


Attrition occurs when an outcome variable is not avaiiabie for aii participants initiaiiy assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 12. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 12. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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